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Abstract — We consider a new fundamental question re- 
garding the point-to-point memoryless channel. The 
source-channel separation theorem indicates that random 
codebook construction for lossy source compression and 
channel coding can be independently constructed and 
paired to achieve optimal performance for coordinating a 
source sequence with a reconstruction sequence. But what 
if we want the channel input to also be coordinated with 
the source and reconstruction? Such situations arise in 
network communication problems, where the correlation 
inherent in the information sources can be used to correlate 
channel inputs. 

Hybrid codes have been shown to be useful in a number 
of network communication problems. In this work we 
highUght their advantages over purely digital codebook 
construction by applying them to the point-to-point setting, 
coordinating both the channel input and the reconstruction 
with the sourceQ 

Introduction 

In point-to-point communication, one is usually con- 
cerned with minimizing the distortion between the source 
sequence and the reproduction sequence S*". In 
common parlance, we desire necessary and sufficient 
conditions such that E[d(S'", 5"")] < D. In the language 
of coordination Q, we seek necessary and sufficient con- 
ditions such that the empirical distribution of (5", S^) 
is close to a given Pg^ in total variation. Both of these 
notions consider the source sequence and reproduction 
sequence, but fail to include the channel input and output 
sequences, X" and Y^. We wish to understand the nature 
of optimal communication in scenarios where we might 
want X" or Y"- to be correlated with (or, coordinate 
with) or S"*. To that end, we investigate which em- 
pirical distributions of (S", X", y", S"") can be reliably 
achieved if we design our encoder and decoder optimally. 
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Fig. 1. Point-to-point communication. An information source S" 
is encoded for communication through a noisy channel Py\x- We 
consider how the channel input and output, X" and Y", can be 
correlated with the source and reconstruction, 5*" and S". 

By understanding which empirical distributions can be 
achieved in the point-to-point communication problem, 
including the channel inputs and outputs, we gain in- 
sights that extend to network communication problems 
involving multiple correlated sources. When information 
sources are correlated and communication occurs over 
channels with interference, it is useful to take advantage 
of the correlation and use it to strategically correlate the 
channel inputs to minimize the effect of the interference. 
The point-to-point setting gives us a simple environment 
to explore the extent with which this is possible and the 
encoding and decoding techniques needed. 

Despite the simplicity of the point-to-point setting, some 
interesting technology applications arise. For example, a 
media broadcast, such as video, which is designed to 
accommodate both analog and digital receivers, fits well 
into this framework. The most familiar approach in this 
case is to split the communication resources (bandwidth 
or power) into an analog transmission and a digital 
transmission, and to take advantage of the analog signal 
as side information for digital signal (as in the Wyner-Ziv 
setting). However, it is not necessarily optimal to split the 
communication resources. A single signal can serve both 
purposes, and in the context of coordination we want 
the channel output y" and the digital reconstruction 5" 
to each satisfy different distortion constraints. Another 
interesting application is the design of "systematic" 
transmissions, where an analog representation of the 
source information is designed to be robust to noise. This 
fits into our framework - we simply ask for coordination 



that results in the reconstruction 5" being equal to 
the channel input X'^. Finally, digital watermarking can 
also be studied in this coordination framework. In this 
case, the watermarked media is the channel input, which 
should obviously be highly correlated with the original 
media content. Additionally, the watermark (5*") should 
be detectable even if the media is altered in a limited 
way. 

A classic result in information theory is the source- 
channel separation theorem, which says that if we have 
designed the optimal source codec and channel codec, 
then we can minimize average distortion by concate- 
nating the two operations, using "bits" as glue. For 
our problem, however, separation is not good enough. 
We draw on recent results on hybrid codes to strictly 
improve the optimal regions. Hybrid codes combine bits, 
which are digital entities, with symbol-by-symbol oper- 
ations, which are analog entities. These analog/digital 
combinations have been studied recently in the context 
of joint source-channel coding for discrete memoryless 
sources and channels and have been applied to multiuser 
scenarios in yj and L2J- 

Our problem is closely related to state amplification ||3l. 
There, 5" is again given by nature, but the channel is 
given by Py\xs^ 1^°^ Py\x- The receiver's goal is to 
reproduce the state sequence, in contrast to the Gel'fand- 
Pinsker problem where the receiver wants to identify the 
sender's message. To use such a channel, X" must be 
coordinated with 5""; hence the relevance to our problem. 
The relation to state amplification and the Gel'fand- 
Pinsker problem prompts us to also consider the scenario 
where the encoder is constrained to be causal. That 
is, the input to the channel can only depend on past 
source symbols, as opposed to depending on the entire 
source sequence. In both this case and the strictly causal 
case, we identify the coordination region, using methods 
similar to those used in the analysis of causal state 
amplification in [4J and i5J- 

Setup 

We are given an i.i.d. source parametrized by Ps and 
a memoryless stationary channel parametrized by Py\x- 
The alphabets for source symbols, channel inputs, and 
channel outputs are denoted by S, X, and 3^, respec- 
tively. In accordance with Figure [T] we have at our 
disposal an encoder and decoder; depending on the 



scenario, the encoder might be noncausal, causal, or 
strictly causal. For block length n, the encoder and 
decoder together constitute a code {f^,g'^) where /" 
and (7" are sequences of functions {/i}r=i ^'i^ {9i}^=i- 

• noncausal encoder: fiiS^^ X. 

• causal encoder: fi-.S"^^ X. 

• strictly causal encoder: /j : — >• X. 

• decoder: gi : y" — )• S. 

In the three cases above, at time i the encoder produces 
the channel input Xi. After n uses of the memoryless 
channel, the (noncausal) decoder produces 5" according 
to Si = QiiY^). The actions of the encoder and decoder, 
combined with the nature of the source and channel, give 
us an induced distribution on (5", X", F", S"). 

Our model being defined, we can now state the condition 
for achievability. 

Definition 1. A distribution PgxYS achievable if for 
all e > there exists a block length n and a code 
if" J 9") such that 

{\\^S"X"Y"S" ~ ■^SXYs\\tV ^) ^ ^' 

where Pg„X"Y"S" empirical distribution of the 

tuple (S"", X", y", 5") induced by the chosen code. 

It's not too hard to see that an achievable PgxYS ^^^^ 
necessarily factor like PsPx\sPy\xPs\sxy' where Ps 
and Py\x ^£ the given source and channel parameters. 
This will be the case because Ps and Py\x ^£ given 
by nature, and by the law of large numbers they will be 
close to the empirical distributions and Py,>|X" for 
sufficiently large n. 

We can view proximity in total variation as akin to 
typicality. 

Definition 2. Given a distribution Pxy, the pair 
is in the e-typical set Te{X^Y) if 

\\Px-^,y'^ — PxyWtv < e 

where 

1 " 

Pr",y" {x, y) = - ^l{xi = x,yi = y} 

^ i=l 

is the empirical distribution of (x", y"). 

This is not too far from the usual definition of strong 
typicality. With this definition, we see that achieving 



PgXYS nothing more than ensuring that, Ve > 0, 
(-^n^^n yn ^.n) ^ Te{SXYS) with high probabiUty. 
Next we define the coordination region, which will be 
the focus of our results. 

Definition 3. The coordination region C is the set of 

achievable PgxYS' 



It is not hard to prove that C is a closed, convex set. 



Theorem 3 (Strictly causal encoder). Let Ps and Py\x 
be the given source and channel parameters. With strictly 
causal encoder, the coordination region C is given by 
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SXYS ■ 

3V s.t. 

PsxYsv = PsPxPv\xsPy\xPs\vy^ 
{ I{X,V;S)<IiX,V;Y). 

Additionally, Pg^yy ^^'^ t>e restricted to be a function 
s{v,y) instead of a general conditional distribution. 



Results 



We begin with an inner bound on the coordination region 
for the case of noncausal encoder. 

Theorem 1 (Noncausal encoder). Let Ps and Py\x 
be the given source and channel parameters. When the 
encoder is allowed to be noncausal, an inner bound for 
the coordination region C is given by 
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^SXYS 

3U s.t. 



^SXYSU - PsPu\i 
I{U;S)<I{U;Y). 



Px\usPy\xPs\uy^ 



cC 



Additionally, Px\su '^^^ ^s\UY '^'^^ restricted to func- 
tions x{s,u) and s{u,y) instead of general conditional 
distributions. 



Example 



Now we consider an example of what can be achieved 
using hybrid codes that is not achievable using Shannon's 
separation method. When the encoder is noncausal, ap- 
plying the separation method yields the following set of 
achievable distributions: 



P. 



SXYS 



^sxYS - PsPs\sPxPy\x 
I{S;S)<I{X;Y) 



cC 



In particular, achievable PgxYS required to factor as 
Pg^PxY- In contrast. Theorem 1 allows for greater range 
of correlation among S, X, Y, and S; the following is 
an example of this. 



This inner bound is a by-product of the achievabiUty 
scheme for hybrid codes (see |1|,[2|). For more discus- 
sion on this, see the achievability section below. 

The next two theorems give a complete characterization 
of the coordination region when the encoder is causal or 
strictly causal. 

Theorem 2 (Causal encoder). Let Ps and Py\x 
the given source and channel parameters. With causal 
encoder, the coordination region C is given by 

( P ■ 

^SXYS • 

^ ^ I 3U,V S.t. 

^sxYsuv = PsPuPv\suPx\suPy\xPs\uvy^ 
{ I{U,V-S)<I{U,V;Y). 

Additionally, Px\su '^^d Pg^uyy '^'^^ restricted to 
functions x{s,u) and s{u,v,y) instead of general con- 
ditional distributions. 



Let Ps be Bern(p) and Py\x be BSC(e). Consider a 
distribution Psxys ^^^^ consistent with Ps and Py\X' 
and also satisfies F[S ^ S] = F[S ^ X] = d. In other 
words, we want our reproduction S and our channel 
input X to be close to S in hamming distortion. Notice 
that these conditions do not fully specify the distribution. 
We will now see that if e < d < p < ^, Theorem 1 
guarantees achievability of Psxys' niake no claims 
here of necessity, just sufficiency. 

To show the claim, choose X such that P[S' 7^ X] = d 
and I{X;S) is minimized. From binary rate-distortion, 
we know this means X ~ Bern(^). Choose U = X 
and S(U, Y) = X. With these choices, we see that 

^sxYsu = PsPu\sPx\usPy\xPs\uy 

because ¥[S ^ S] = ¥[S ^ X] = d. Now we just 
need to check that I{U;S) < I{U]Y). We can view 
our situation as S = X (B Zi and Y = X (B Z2, where 



Z\ ~ Bem((i) and Z2 ~ Bem(e) are independent of X. 
Then we have 

/(C/;5) = 1{X-S) 

= H{S)-H{Zi) 
< H{Y)-H{Z2) 
= I{X;Y) 
= I{U;Y). 

where H{Z2) < H{Zi) is due to e < d and H{S) < 
H{Y) is due to the fact that the entropy of a random 
variable increases when passed through a binary sym- 
metric channel, in this case BSC((i*e). 

ACHIEVABILITY 

The proof of achievability for the case of noncausal 
encoder, obtained by using hybrid codes, comes directly 
from m, so we will omit it. However, an illustration of 
hybrid codes is depicted in Figure [2] The main idea is 
to find an auxiliary [/" sequence that is jointly typical 
with the source S" and is part of a sparse codebook that 
covers 5"^. Then the channel input is constructed symbol- 
by-symbol as a function of S and U. This symbol- 
by-symbol function (the analog part) allows for X" to 
be correlated with 5", while the construction of the 
auxiliary sequence (the digital part) takes advantage of 
compressing the entire source sequence. The decoder 
obtains f/" after observing Y^ due to the sparsity of 
the codebook and then constructs 5" symbol-by-symbol 
from both Y and U. 

The achievability schemes for Theorems 2 and 3 are very 
similar to H and ||5l, and we will now provide a sketch 
of the proof when the encoder is strictly causal (Theorem 
3). 

Fix Px, Pv\xs ^iid s{v,y) consistent with an interior 
point of C. We will use block-Markov-like encoding, 
viewing the source as sequence of blocks: 

The idea is to compress S^{i — 1) in the zth block, 
using — 1) and Y^{i — 1) as side information. 

We first cover the source by using at least 2"^('^'^l"'^) 
codewords and assign them randomly to greater than 
2ni{S;V\x,Y) ^^^^ ^-^^ gj-gj Qj-jgj- [j^ (j^g exponent). During 

the ith block we identify an index £i such that V^{£i) 
is jointly typical with S^{i — 1) and X^{i — 1) and 




Fig. 2. Hybrid code encoder. The encoder for a hybrid code has two 
steps in producing channel inputs form a source sequence 5*". First, 
the source sequence is compressed using lossy compression and an 
auxiliary sequence C/". Then, the channel inputs X" are constructed 
symbol-by-symbol from and 17". If properly designed, the 
decoder is then able to decode [/" from and use both sequences 
to form the reconstruction 5*". 

then transmit X"'{mi), where rrii is the bin index of 
V^{ii). If the number of bins does not exceed 2"^^"'*-'^) 
and the size of each bin does not exceed 2"^(^'^l^\ 
then the decoder will be able to recover £i by using 
— 1) and Y"-{i — 1) as side information. Finally, 
the decoder produces S^{i — l) symbol-by-symbol using 
s{v, y). Under the standard random coding analysis, this 
scheme ensures joint typicality with high probability as 
long as /(X, V; S) < I{X, V; Y). 

When the encoder is causal (but not strictly causal), the 
proof is similar, except now we introduce an additional 
auxiliary random variable U that plays the role of X 
in the strictly causal case. The channel input X" is 
produced symbol-by-symbol with a function x(u, s). 

Converse 

The converses for Theorems 2 and 3 are similar; we 
show the latter here. To prove the converse for Theorem 
3, it is enough to show that for all e > 0, there exists a 
distribution 

^sxYSV = PsPxPv\xsPy\xPs\vy 

such that 

. I{X,V;S)<I{X,V;Y) 

• W^SXYS ~ ^SXYsW"^^ ^ ^ 

Then by the closedness of C we will be done. The key to 
the proof is in the identification of the auxiliary random 



variables. We first introduce Q _L (S'^, X", F", S") 
uniformly distributed on the set {l,...,n}, and define 
Vq = {S^~^,Y^^^). Then we define P by setting 
iS,X,Y,S) = iSQ,XQ,YQ,SQ) and V = {Vq,Q). 
With these choices, we can verify that S -L X, SV — 
X -Y, and SX -YV -S all hold. Finally, we have 

I{X,V;S) = I{Xq,Vq,Q;Sq) 
= I{Xq,Vq;Sq\Q) 

n 



n 



n ^-^ 

1=1 

n 
i=l 

i=l 

1 " 

1=1 

< -y^i(x,,Vi-Yi) 

Ti ^ ^ 



= 1{Xq,Vq-Yq\Q) 
<IiXQ,VQ,Q;YQ) 
= I{X,V;Y) 

To finish the proof, we can use our hypothesis that 
PgxYS achievable to show that 

Hp ' — p - II < F 

W-^ sxYS ^sxysWtv ^ 

The details of such a claim can be found in f6l, but the 
key is that 



hybrid analog-digital codes are optimal when the encoder 
is causal or strictly causal. 

We find it interesting that we need not look further than 
the point-to-point memoryless communication setting 
to find use for unconventional and somewhat complex 
coding schemes such as hybrid codes. 
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Summary 



Joint source-channel coding can be crucial, even in a 
point-to-point memoryless communication setting, when 
the right questions are asked. The question we investigate 
is how strongly the channel input and output can be 
correlated with the information source during a commu- 
nication transmission. Our work makes use of the hybrid 
codes studied in [IJ and [2J and finds that variations of 



